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IDENTIFIERS 



ABSTRACT 



The Beginning Teacher Evaluation Study^ Phase II, was 
a research project on effective teaching behavior — what teachers do 
that significantly affects what and hov pupils learn. The purposes of 
Phase II vere to (1) develop an assessment system for measuring 
teacher and student behaviors and other factors which could influence 
each of them and their interrelationships and (2) generate hypotheses 
about the interrelationships among teacher and pupil behaviors and 
related factors. Subjects were 41 second grade and 54 fifth grade 
experienced teachers in eight school districts in California. Teacher 
reports of their instructional activities and procedures were used as 
one source of descriptive information about classrooms. Different 
kinds of "scores'' from these reports were employed as measures of 
teaching performance for the analyses done in the study. (RC) 
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THE DIARIES AND OVERVIEWS: BACKGROUND 

The overall purpose of Phase II of the Beginning Teacher Evaluation 
Study was to examine the "relationships between the classroom behavior 
of elementary school teachers and the outcomes achieved by students in 
their classrooms." Phase II of the study was devoted to the generation 
of hypotheses and instruments V7hich could be useful in that effort. The 
structured Work Diaries were intended to provide a vehicle for teachers 
to describe their programs in reading and mathematics directly. We were 
interested in whether or not teacher reports of their instructional 
activities and practices would be accurate and valid sources of informa- 
tion about classrooms. 

The Work Diaries were one of several approaches to describing 
teaching performances in the project. The study of teaching performances 
was also addressed by direct observation and by videotaping in the 
classrooms. Two types of "Teaching Performance" variables were derived 
from direct observations in classrooms, observations gathered with a 
behavior recording system and a category system. The third type of 
Teaching Performance variables was derived from the Work Diary scores. 
Scores were aggregated from the several instruments teachers used to 
describe their instructional activities. The data were provided from 
five instruments: 

1. The Reading Program Overview was intended to provide a 
summary of the teacher's reading program for the entire 
school year. It was completed once by teachers. 

2. The Reading Work Diary was designed to reflect the 
instructional process during one school week. The diary 
was completed twice. 
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3. The Mathematics Program Overview was Intended to provide a 
summary of a teacher /.s^^math^feSiatics program for the entire 
school year. It, like the Reading Program Overview, was 
completed once by the teachers. 

4. The Mathematics Work Diary was parallel to the Reading Work 
Diary and was designed to reflect the instructional process 
during one school week. The Diary was completed twice at 
the same time that the Reading Diary was used. Both were 
completed during two target weeks which were eight calendar 
weeks apart. 

5. The Reading and Mathematics Work Diary for Paraprof essionals 
and Other Teaching Adults in the Classroom also covered a one 
week time period and was Intended to describe the functions 
of adult assistants in the classroom. It was completed twice 
by each adult assistant in a classroom at the time the Reading 
and Mathematics Work Diaries were completed by teachers. 

The information from the battery was organized to Indicate, by grade 
level and within subject areas, the academic emphasis, the quality of the 
teaching methodology, the complexity of the organizational structure, and 
the variety and types of instructional materials used. 

Because the instruments were intended for teachers to use with little 
or no direct project staff support, it was essential that both the 
structure and content be understandable and sensible to the teachers. 
Teacher review was an integral part of developing the Diaries. Two 
different groups of participating teachers reviewed the Reading Diary at 
different stages of the developmental process. 
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Teachers were asked to complete the Reading and Mathematics Diaries 
twice during the spring of the data gathering year. Ninety-four of the 
95 teachers returned the first set of Diary materials, and 93 returned 
the second set. Eighty-seven Diaries from other teaching adults were 
returned. 

RESULTS 

Teacher's Descriptions of Their Activities 
and Pupil Outcomes 

The information was organized to indicate, by grade level and within 
subject areas, the academic emphasis, the complexity of organizational 
structure, and the variety and types of instructional materials used. 
The accuracy of sections of the teacher reports was tested against 
observational information from the project classrooms and related to 
pupil skill growth in rc=iading and mathematics. 

Academic Emphases 

Academic emphases included three general information categories: 
how much time teachers reported spending in preparing for and teaching 
reading and mathematics skills; how frequently teachers reported teaching 
specific reading and mathematics skills; and, the specific skills in 
reading and mathematics that teachers reported they did not plan to 
teach. Some of the more interesting results were: 

• Second grade teachers spent more time preparing for and 
teaching reading than did fifth grade teachers. 

• Fifth grade teachers spent more time preparing for and 
teaching mathematics than did second grade teachers. 

• The amount of total teacher preparation time had no 
relation to pupil skill growth in mathematics and an 
inverse relation to pupil skill growth in reading. 
Greater amounts of preparation time were associated 
with poorer reading comprehension skills for fifth 

. grade pupils. ^ 
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• The amount of direct teaching time had no relationship 
to pupil skill growth in mathematics but did relate to 
second grade pupil skill growth in decoding. 

o There was greater grade level variation in time spent 
teaching specific reading skills than there was in the 
teaching of specific mathematics skills. 

• Second grade teachers, as would be expected, spent more 
absolute and proportional time teaching decoding and 
vocabulary skills than did fifth grade teachers. 

• Both second and fifth grade teachers spent more time 
teaching more generic reading comprehension and reading 
application skills than they did teaching more specific 
decoding and vocabulary skills. 

• Greater amounts of time spent teaching the more 
specific decoding and vocabulary reading skills to 
fifth grade pupils was associated with poorer pupil 
performance on the more generic reading tasks, com- 
prehension application and achievement. 

o The time second. grade teachers reported spent in teaching 
specialized skills in mathematics was negatively related 
to pupil growth in computation skills, and positively 
related to concept skill growth. 

The time teachers spent teaching specific reading and mathematics 
skills in relationship to pupil performance suggests a variety of research 
tasks which might focus on the identification of optimal ranges of time 
for teaching specific skills to pupils in classrooms with a variety of 
skill configurations. Our data suggest, for example, that fifth grade 
teachers who emphasize basic decoding and vocabulary skills during reading 
instruction may do so at the expense of pupil skill growth in comprehension 
skills. The time teachers reported spending in preparing for and teaching 
reading or mathematics skills were the basis for the combined category, 
WD~1. 

The quality of the teaching methodology . Teachers were asked to 
describe the teaching strategies, pupil activities, and materials used 
both for skills they introduced and reviewed during a given week. Their 
responses were rated. The "scores" became the combined category 
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Neither score by itself related to pupil skill growth at the second grade 
level; one related to pupil skill growth in fifth grade reading. 

The repor ted frequency of reading skills taught . Teachers were asked 
to check skills that they had "introduced" or "reviewed" during each of 
the two weeks the Diaries were maintained, in other words, skills they had 
actually taught. Teachers were also asked to indicate in the Program 
Overviews which specific skill*> would not be taught during the school year, 
either because "pupils already know these skills" or "these skills are 
taught at a higher level." The frequency of skills taught and not taught 
provided the basis for the combined category score \^-2. 

• The number of skills taught bad little proportional 
relationship to the time spent teaching within a skill 
area. Second grade teachers, for example, reported 
spending about twice as much time teaching decoding 
skills as did fifth grade teachers. The numbers of 
actual skills taught, however, do not appear that 
different (ten skills and eight skills) . 

• As was the case in time spent, there was greater grade 
level variation in number of skills taught in reading 
than there was in mathematics. 

But where the number of skills taught differed it made a difference 
in pupil skills, particularly in mathematics. 

• The number of specific skills taught related to pupil 
skill growth in mathematics at both grade levels, and 
was particularly important for second grade pupils. 

o The member of teachers who did not plan to teach 

specific skills was relatively small in both reading 
and mathematics. But the decision not to teach 
certain skills related to pupil outcomes in several 
instances. For example, the decision not to teach 
certain decoding and vocabulary skills by a relatively 
small proportion of second grade teachers was related 
to the poorer decoding and vocabulary skill growth of 
their pupils. 



- 5 - 



The Complexity of the Organizational Structure in the Classroom 



Organizational structure included t^^o general type^s of information 
about the project classrooms. Teachers reported how they grouped 
children for reading and mathematics instruction, and what proportion 
of the pupils were taught reading or mathematics only by themselves. 
The ratio of teaching adults to children was also determined for both 
subject areas. These variables became the basis for the combined 
category of teaching performances called 

• Pupils were grouped differently for reading and 
mathematics instruction. Simultaneous instruction of 
all pupils was the most common pattern used for mathe- 
matics instruction, and grouping by general reading 
level wias the most common for reading instruction. 

o Grouping practices did not relate to fifth grade 

pupil achievement in mathematics. However, they did 
relate to several kinds of pupil achievement in 
reading and in second grade mathematics. For example, 
grouping by general reading level, the most common 
convention, related positively to second grade pupil 
skill growth in application skills and negatively to 
decoding skill growth for fifth grade pupils. 

a Few teachers reported individualized instruction as 
their preferred organizational mode. 

o Individualized reading instruction was associated with 
poorer second grade pupil performance on reading 
application tasks, but individualized mathematics 
instruction was associated with second grade pupil 
growth in mathematics skills. 

9 More fifth grade pupils were taught reading and 
mathematics exclusively by their teacher vthan were 
second grade pupils. 

• Second grade teachers utilized up to six teaching adults 
during reading instruction, and up to five during 
mathematics. Fifth grade teachers who shared teaching 
or preparation responsibilities did so with a maximum 

of one or two other adults. 

e Pupil skill grox7th in reading; particularly that of 

second grade pupils, was associated with the proportion 
of pupils taught exclusively by their teacher. No such 
association was evident for mathematics instruction. 
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The generalization that the percentage of pupils taught exclus:'.veiy 
by the teacher related to reading skill growth at both grade Xf>vels was 
an intriguing finding. A variety of /teaching configurations was 
represented in the sample: team teaching, multiple level, nongraded and 
open classrooms. A variety of human and school resources was available 
to some teachers: aides, student teachers, volunteers, reading special- 
ists and learning laboratories. Under the circumstances (particularly in 
the case of the younger pupils) the simple generalization that pupil 
skill growth in reading may relate to the proportion of pupils taught 
exclusively by the teacher should be studied further. Although the data 
gathered from the paraprof essionals and other teaching adults was not 
detailed for this report, it is of considerable interest to note that the 
Work Diary variables associated with other teaching adults were consistently 
and significantly related to lack of pupil growth in reading and mathe- 
matics skills. 

The Variety and Types of Instructional Materials 
Used in Reading and Mathematics Programs 

Teachers were asked to approximate the percentage ot their reading 

and mathematics programs in which they used the types of materials listed. 

The teachers were provided with a list of twelve different types of 

reading materials and ten different types of mathematics materials. 

Space was provided to specify the exact materials used and to indicate 

any types not included in the lists. 

• Patterns of reading materials usage varied between 
second and fifth grade. Second grade teachers, for 
example, used basal readers more than did the fifth 
grade teachers, who relied more on individualized 
materials and reading kits. 

• There was little percentage difference between second 
and fifth grade usage of the ten types of mathematics 
instructional materials. 



• The use of the state adopted reading basal textbook 
series was associated with second grade pupil growth 
in application skills. 

• The use of the state adopted mathematics basal textbook 
series at fifth grade was associated with poor pupil 
performance in computation skills • 

o The use of programmed materials was associated with 
second grade pupil growth in computation and total 
tnathematics skills. 

The materials used in reading and mathematics instruction were 
aggregated somewhat differently to provide the basis for the combined 
category W)-5. There the number of different kinds of materials used 
became the index for 

Teacher Reports of Their Activities and Observations 
of those Activities 

The validity of teacher reports of i.heir activities was explored in 
comparison to selected observational variables. The validity of teacher 
reports considered were: 

1) Teacher reports about the organization of pupils for reading 
and mathematics instruction was compared with observed 
classroom organizational patterns; 

2) Teacher reports of time spent teaching specific reading and 
mathematics skills was compared with observed incidence of 
positive pupil learning; and, 

3) Teacher reports of instructional activities, grouping and 
materials used was compared with pupil achievement in 
reading and mathematics. 

Our purpose was to determine if reports provided by teachers related 
to other information obtained in the study. 
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Observed and Teach er Reported Organization oi: Pupils for Learning 

One of the two observational systems utilized to gather Phase II 
data was APPLE* (Anecdotal Processing to Promote the Learning Experience), 
a system in which observers generated a low inference record of teacher 
and pupil behavior. Among the information observers recorded was a 
continuous record of instructional contexts in which the sample of 
target pupils were working during jreading acd mathematics instruction. 
In the APPLE System: 

...The concept of an instructional context defines 
the relationship of the teacher to the instructional 
activity of the pupil. There are six major contexts, 
ranging from the teachftr directing the activities of 
the entire class to an individual student working or 
playing independently of the teacher's direction and 
independently of other children in the class. 
(Volume III, Chapter 1, p. 9) 

For the analyses, only those observer descriptions of pupils 
specifically working in reading or in mathematics contexts were included. 

Eight target pupils were identified for observation in each class- 
room. Pupils and appropriate alternates were selected by comparing 
teacher rankings of expected pupil performance with the actual fall 
reading performance on one or more of the Phase II tests. A boy and a 
girl were selected from the top, from just above the middle, from just 
below the middle and from the bottom of the distributions in each 
classroom. The list was also balanced to reflect the ethnic composition 
in each classroom. The analyses in this section of the report compare 
the observed activities of that sample of less able, average and above 
average pupils in each classroom with teacher reported acitvities. 



*A1'PLE was developed by Nadine Lambert of the University of California 
Berkeley. It is described in its application to this project in Chapter 
One of Volume III of the final report. 
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Teachers were asked to indicate in the Work Diaries which one 
organizational pattern most nearly described their current reading program* 
Their choices were: simultaneous instruction of all pupils, pupils 
grouped by reading level, pupils -grouped by skill needs, individualized 
instruction and an open-ended "other" category* The four specific 
organiEational patterns reported are compared with the six instructional 
contexts in which pupils were observed to be working. 

The first indicator we have examined of the validity of a teacher's 
reports of his or her own activities was productive. Observed and teacher 
reported organization of pupils for learning seems to be consistent to the 
degree that further research would be warranted to refine systems for 
teachers to report their o\m organizational strategies* 

An Interesting aspect of this analysis was that during reading and 
mathematics activities at both grade levels adults other than the teacher 
were observed working in classrooms where children were grouped instruc- 
tionally for specific skill need. Only about one-- third of the second 
grade teachers and about fifteen percent of the fifth grade teachers did 
not teach reading and mathematics to all of their pupils themselves. 
Therefore, V7e are discussing a relatively small proportion of the teachers. 
Teachers were not observed (to a degree where significant correlations 
emerged) teaching groups of pupils in classrooms where they reported 
grouping by skill level* It appears that teachers who use other adults 
as instructors frequently assign the major repsonsibility for specific 
skill instruction to the other teaching adults. We have previously 
suggested that instruction associated with other teaching adults was 
consistently and negatively associated with pupil achievement. 



Observ ed Incidence of Specific Kinds of Positive Pupil Learning 
and Te acher Reported Time Spent Teaching Related Reading and 
Mathematics Skills ' 

Teachers estimated the amount of time they spent teaching four kinds 
of reading skills: decoding, vocabulary, comprehension and application. 
APPLE observers rated, as well as recorded, pupil activities during reading 
and raathematics instruction. Of the pupil "events" which occurr,?.d 
frequently enough to be included in the APPLE analytical phases of the 
study, it appeared that two of the reading event categories might relate 
to two of the Work Diary categories. To test those assumptions the 
observational evidence of positive pupil performance in the demonstration 
of phonics skills was compared to the Work Diary reports of the time each 
teacher spent teaching decoding skills; and, the observed evidence of 
positive pupil performance in reading comprehension was compared to reports 
of time spent teaching comprehension skills (Table 18) . 

The APPLE lexicon definition of a "+ phonics skills" pupil event, 
for example, is "positive evidence of skill in sounding out new words; 
knowing the sound connected with letters." If observers had any doubt 
about the quality of a pupil activity, they were trained to rate it as 
a "0" or neutral event. Only the "+" or positive incidences of pupil 
performance were used in this analysis. 

The observed incidence of positive pupil performance during mathe- 
matics in number concept skills was compared to the two potentially 
related Work Diary categories, the time Reported teaching problem types 
and the: time reported teaching operations (Table 19). At the fifth grade 
level the correlations do suggest that teacher reported amounts of time 
spent in direct instruction of specific reading and mathematics skills 
was related to observed evidence of successful pupil performance in those 
skill areas. 
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The second test of the validity of teacher reports was a comparison 
of the reported time spent teaching specific reading and mathematics 
skills compared with observed evidence of pupil demonstrations of those 
skills. The correlations were small and the levels of significance 
approached the liberal .10 level arbitrarily set by the author as the 
cut off point for these analyses s, the reports of teachers 

about time spent teaching spec-^ lI- ppeared to be reflected In 

evidence of pupil skills in those areas. 

Teacher reports of their instructional activities as they related to 
pupil skill growth have been discussed in the previous section. 

Teacher reports of their instructional activities, procedures and 
materials have demonstrated their utility and quality in this study. The 
Phase II teachers who expended so much effort in providing that information 
should be gratified to know that the results justify their efforts. "Self- 
reporting" as a reputable form of gathering research information may have 
gained new lustre as a result of their careful efforts. 
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